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Abstract 

Suppose that T is a continuous and self-adjoint Hankel operator on L 2 (0, oo) with 
kernel <p(x + y) and that Lf = ~^( a ( x )-£;) + b( x )fi x ) with a(0) = 0. If a and b are 
both quadratic, hyperbolic or trigonometric functions, and </> satisfies a suitable form of 
Gauss's hypergeometric differential equation, or the confluent hypergeometric equation, 
then FL = LF. The paper catalogues the commuting pairs F and L, including important 
cases in random matrix theory. There are also results proving rapid decay of the singular 
numbers of Hankel integral operators with kernels that are analytic and of exponential 
decay in the right half-plane. 
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1. Introduction 

For G L 2 (0, oo), we recall that the Hankel operator with kernel <fi(x + y) is the 
integral operator 

n OO 

F^f(s)= / <P(s + t)f(t)dt (1.1) 



X 



JO 

from a subspace of L 2 (0, oo) into L 2 (0, oo). 

Megretskh, Peller and Treil [6] determined the possible spectral multiplicity function 
that a continuous and self-adjoint Hankel operator can have; however, their spectral theory 
does not yield much information about the eigenvectors. Tracy and Widom observed that 
some self-adjoint and compact Hankel operators commute with self-adjoint second-order 
differential operators L that have purely discrete spectrum, and hence and L have 
a common ort ho normal basis of eigenfunctions; see [9, 10]. One can then use the WKB 
approximation to describe the asymptotic eigenvalue distribution of L, and the asymptotic 
behaviour of the eigenfunctions. 
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In this paper we start with L and seek choices of <f>, thus reversing the order of steps 
in [9, 10]. Suppose that 

df 



(1.2) 



where a and b are three times differentiable functions such that a(0) = 0. Suppose that 
a'" = Aa! and b'" = Bb\ for some constants A and B. In section 2 we derive an explicit 
differential equation 

4>"{u) + a{u)<j)'{u) - 0(u)<j>(u) = (1.3) 

which ensures that T^L = LT^. 

The possibilities for a and b depend upon the sign of A, and in sections 3, 4 and 
5 we consider the quadratic, hyperbolic and trigonometric cases in detail. In all cases, 
the differential equation reduces to a linear differential equation with rational functions as 
coefficients which has less than or equal to three singular points, and we determine the 
nature of the singularities. Thus we prove the following theorem. 

Theorem 1.1. The differential equation (1.3) may be transformed by change of variables 
to the hypergeometric equation 



/-, \ d 2 4> r . , ., x -, dd> 

x{l - x )-^ + {A - {p + v + l)x\— - 







(1.4) 



or confluent hypergeometric equation. 

This result covers cases relating to standard models in physics. In [11] Tracy and 
Widom considered the integral operators that have kernels of the form 



W(x,y) = 



f{x)g{y) - f{y)g{x) 

x-y 



(1.5) 



where / and g satisfy 



"/(*)" 




p(x) 


q(x) 




'fix) 


_g(x)_ 




—r(x) 


—p(x) 




_g{x) 



(1.6) 



for some real polynomial functions p(x),q(x),r(x) and m(x). Of particular interest are 
the cases where m(x) = l,x or 1 — x 2 . The main application of Tracy-Widom operators 
W is in random matrix theory, where they describe the eigenvalue distributions of n x n 
random matrices from the generalized unitary ensemble as n — > oo. 

Tracy and Widom introduced an indirect process for computing the spectrum of W, 
and applied it effectively to the Airy and Bessel kernels which are fundamentally important 
cases in random matrix theory; see [9, 10]. Specifically, the edge distribution is given 
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by det(I — WP( XjOQ )), where P( x ,<x) is the orthogonal projection L 2 (0, oo) — > L 2 (x,oo) 
for a; > 0. Their first step was to introduce a self-adjoint Hankel operator T such that 
T 2 = W, then to use the spectral theory of Hankel operators to deduce information about 
the spectrum of W. 

The identity W = V 2 is equivalent to the factorization of kernels 



f{x)g{y) - f{y)g{x) 



X 



/•oo 

= / <j){x + t)<j){y + t)dt. (1.7) 
Jo 



Now is Hilbert-Schmidt if and only if J °° t\<f>(t) \ 2 dt < oo. If 1^ is Hilbert-Schmidt, 
then is of trace class, and det(J — T 2 ) is defined. 

In [2, 3] we considered several differential equations of the form (1.5) and resolved 
whether factorization of W takes place, giving explicit formulae for 0; several important 
examples involve the confluent hypergeometric equation, which may be reduced by change 
of variable to Whittaker's equation. For a more general statement about reducing systems, 
and how changes of variable can affect factorization, see [3, section 3]. 

By carrying out the reduction in Theorem 1.1 explicitly, we recover the Tracy-Widom 
kernels as special cases of a more general theory, and also obtain new examples of com- 
muting r<£ and L; there remain a few residual cases that we have not been able to solve 
explicitly in terms of standard functions. 

In section 6 we present results concerning the rate of decay of the singular numbers 
of T under hypotheses that are well suited to applications in random matrix theory. 

We follow standard notation as used in [4], and let be the Laguerre polynomial 
of degree n and order a, so 

„ — a jn 

L ( n\s) = ^e° — (e-^+«) ( S >0). (1.8) 

2. The main result 

Theorem 2.1. Let V be the Hankel operator that has kernel 4>{x + y), let L be the 
differential operator 

Lf(x) = ~(a(x)^)+b(x)f(x), (2.1) 

where a"'(x) = Aa'(x) and a(0) = 0. 

(i) Then there exists a real function a such that a(x+y) = {a'{x)—a'{y))/{a{x)—a{y)). 

(ii) Suppose further that f3(x + y) = (b(x) — b(y))/ (a(x) — a(y)) for some real function 
P and that 

<f>"{u) + a(u)(j>'(u) - (3(u)(f)(u) = 0. (2.2) 
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Then the operators V and L commute on C£°(0, oo). 

Proof. Let T t : L 2 (0, oo) — > L 2 (0, oo) be the translation operator T t f(x) = f(x + t) and 
Tj the adjoint for t > 0. Then by the fundamental property of Hankel operators, we 
have T t T = TT^ and hence J^T = — T-^ on C£°(0, oo). To exploit this, we introduce the 
expression 

y) = (a(y) - a(x))(j)"(x + y) + (a'(y) - a'(x))(p'(x + y) - (b(y) - b(x))(p(x + y); (2.3) 
then by successive integrations by parts, we obtain 

(LT - FL)f(x) = <P(x + y)a(y)f'(y) -<f>'(x + y)a(y)f(y) + / y)f(y) dy. (2.4) 

L J Jo 

The term in square brackets vanishes since a(0) = 0; so the operators V and L commute 
if and only if $ = 0. The main idea is to reduce the condition $ = to the differential 
equation 

(a(x) - a(y)) ((f)" (x + y) + a(x + y)<j>'{x + y) - f3(x + y)<j>(x + y)) = 0, (2.5) 

involving functions a and (3 of only one variable u = x+y. The following lemma guarantees 
the existence of such a and f3 under suitable hypotheses, and hence gives the proof of 
Theorem 2.1. 

Lemma 2.2. Suppose that a and b are three times continuously differentiable and non 
constant real functions. 

(i) There exists a differentiable function a such that 

a'(x) — a'(y) 
a(x) - a(y) 

if and only if a"'(x) = Aa'(x) for some constant A. 

(ii) Ifa"'(x) = Aa'(x), then a(x) —a(y) = f(x + y)g(x — y) for some differentiable functions 
f and g. 

(in) Suppose that, for some differentiable functions g and h, 

Then there exists a constant B such that b"'(x) = Bb'(x). 
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Proof, (i) (=>•) We have 

Vox aj// V a(x) — a(j/J / 

= (a // (x)a(x) - a'Qr) 2 ) - (a"(y)a(j/) - a'(y) 2 ) + (a(^K(y) - a"(x)a(y)) 

(a(x)-a(y)) 2 ' 1 ' J 

so we have 



a 2 / 



(a(x)a"(y) - a"(x)a(i/)) = 0, (2.10) 



dxdy 

which soon reduces to a"'{x) = Aa'(x) for some constant A. 

(i) (<£=) There are three main families of solutions of the differential equation a"'{x) = 
Aa'(x), depending upon the sign of A: 

(Q) quadratic case when A = and a(x) = q 2 x 2 + q\x, with q 2 and q\ not both zero, 

then 

a (u) = 2q2 ; (2.11) 
q2U + qi ' 

(H) hyperbolic case where A > and a(x) = /ii coshte + /i 2 sinhte + /i 3 , with t 2 = A, 
hs = —hi and h\ and /i 2 not both zero, then a(0) = and 

. . thi cosht'u/2 + th 2 sinhtw/2 

= "I r-; 77: ; : 77-; (2-12) 

(C) circular case when A < and a(x) = c\ costx + c 2 sinte + c 3 , where t 2 = —A, 
C3 = — ci, and ci and c 2 not both zero; then a(0) = and 

. . — tc\ costu/2 — tc 2 sintu/2 

a(u) = ; ; — . (2.13) 

— C\ sin tu/2 + c 2 cos tit/2 

(ii) One considers cases (Q),(H) and (C), and applies the addition rule of trigonometry 
or hyperbolic trigonometry to factorize a(x) — a(y). 

(iii) When such g and h exist, we have 

(dx dy) ( g(x — y) ) ^' (^-14) 

which reduces to 

&'(£) + ft* (y) _ 2g'(x-y) 
b(x) - b(y) g(x - y) 

and arguing as in (i), we deduce that b"'{x) = Bb'(x) for some constant B. 

□ 
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Remark. Without loss of generality we can assume that a and b are bounded above or 
below on (0, oo). If a(x) > for all x > and b is bounded below, then the quadratic form 
associated with L is bounded below on C£°(0, oo), and hence L admits of a self-adjoint 
extension by Friedrichs's theorem. 

3. Quadratic cases 

In this section, we determine all the quadratic and linear choices of a and b and the 
corresponding <fi such that L and commute. We look for a and f3 as in Theorem 2.1. 

Lemma 3.1. Suppose that a(x) = q 2 x + q\x + qo- Then (3 of Theorem 2.1 exists if and 
only ifb is quadratic or linear; so that b(x) = bix 2 + b\x + bo, and then 

m = hmh. ( 3.i) 

q2X + qi 

Proof. We need to find (3 and b such that 

h{x) ~ = (<&(* + V) + qiMx + y), (3.2) 
x-y 

so b"'(x) = Bb'(x) by Lemma 2.2. Hence b must belong to one of the types (Q), (H) and 
(C), and evidently by (2.15) only a quadratic has the right form. 

□ 

Proposition 3.2. The differential equation (2.2) for <j) is 

(m) ; <P («) H ; 4>\u) = 0, 3.3 

q 2 u + qx q 2 u + qi 

which may be transformed by change of variables to an elementary or confluent form of 
the hypergeometric differential equation. 

Proof. This formula follows from Theorem 2.1 and Lemma 3.1. The differential equation 
(3.3) has less than or equal to two singular points, the only possibilities being —q\jq2 and 
infinity. Determining the nature of the singularities in all possible cases below, we find that 
cases Q(i) and Q(ii) are trivial; cases Q(iii) and Q(iv) are elementary; in cases Q(iv)-(viii), 
infinity is an irregular singular point; in cases Q(vi)-(viii) —qxjqi is a regular singular 
point. We deduce that as a special case of [12, p. 352] and [4, p. 1084], the solution is 
either an elementary function or may be expressed in terms of confluent hypergeometric 
functions. 

□ 

Examples 3.3. (Quadratic cases). We now carry out a systematic reduction of the (3.3), 
bearing in mind that we wish to have a continuous Hankel operator T^. 
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Q(i) If qi = q2 = 0, then a(x) = and b(x) is constant by (2.3). 

After translating the variable it, we assume henceforth without losing generality that 
exactly one of q\ and q2 is non-zero. 

Q(ii) If q2 = i>2 = b\ = 0, then is linear, so does not give a continuous Hankel 
operator. 

Q(iii) If 6 2 = bi = 0, q2 > and q\ > 0, then 

m = C 1 + — ^— • (3.4) 

In particular, when g 2 = 1> 9i = 0, Ci = and C 2 = 1, we thus obtain Carleman's 
operator T with kernel l/(x + y). Carleman's operator is continuous on L 2 (0, oo) and 
has spectrum [0,7r] with multiplicity two by a result due to Power [6]. Both T and the 
differential operator L = — ^(^ 2 ^) are multipliers for the Mellin transform, so this case 
of Theorem 2.1 was to be expected. 

Q(iv) If q2 = b2 = and b±,qi ^ 0, then (f> is hyperbolic or trigonometric according to 
the sign of b\jq\. In particular, b\ = q\ gives <f>(u) = e~ u , and is a rank-one continuous 
Hankel operator. We defer the case of b\/q\ < until section 5. 

Q(v) If 52 = and 6 2 7^ 0, then we obtain the Airy equation. In particular, if 
a(x) = x and b(x) = x(x + s), then satisfies 4>"(u) = (u + s)(f)(u), so the Airy function 
= Ai(tt + s) gives a solution. Here the associated Hankel operator is Hilbert- 
Schmidt, and L has discrete spectrum as described by Titchmarsh's theory of oscillations 
[8] ; Tracy and Widom discuss the asymptotics of the common sequence of eigenfunctions 
in various ranges [9] . The Hankel operator has square 

M(x)M'(y)-M'(x)M(y) 
x-y 

which is the famous Airy kernel as used in [9] to describe the soft edge of eigenvalues 
distributions from the Gaussian unitary ensemble. 

When qi = 1 and 6 2 < 0, we obtain <p"{u) + u<p{u) = by translation and rescaling, 
and the general solution of this is 

= C.u^J^-u^) +CW /2 J_ 1/3 (^ 3/2 ), (3.6) 
for typical C\ and C 2 with asymptotics [5, p. 133] 

</>(u) xC«- 1/4 cos(V 2 + c) (u->oo). 



Q(vi) If b 2 = and 61,^2 7^ 0, then we obtain after translation and rescaling the 
equation 

V(u)±^0K> = O (3.7) 

with solutions including [5, p. 250] 

0+(w) = w- 1/2 Ki(v^), 0-H = w" 1/2 Ji(v^). (3.8) 
We have by [5, p. 435] 



as u — > 0+ 



<M W )~ < M 1/2 -3/4 -vs (3 - 9) 
If) w ' e v as u — > 00, 

so defines a continuous linear operator on L 2 (0, 00) which is not Hilbert-Schmidt; but 
the compression of to L 2 (x, 00) is Hilbert-Schmidt for x > 0. Further by [5], 

{I as u — > 0+ 

/ \V2 0//1 (3.10) 
ffj w- 3 / 4 cos(v^-37r/4) asw^oo, 

so r<^_ is not Hilbert-Schmidt. As in [3, 2.3], the associated integrable operator is 

x<t>-{x){y<t>-{y))' -{x<t>-{x))'y<t>-{y) 1 f°° 

x-y 4 j 

Q(vii) If 62, 92 7^ 0, then we obtain after translation the equation 

-0» - V(«) + (- + —)(f>{u) = 0. (3.11) 

In particular, when b\ = 0, we can rescale to the equation 

-(f)" (u) --</>' (u) ±Mu) =0 
u 

with solutions 

e" u Kicos« + K 2 sinM 

<M«) = > 0-W = • (3-12) 

u u 

By Theorem 2.1 of [3], with f(x) = e~ x and 

p °° e~ 2t dt 



f 

g{x) = e x 

Jo 



we have 

f(x)g(y)-f(y)g(x) ' 



x-y Jo 



x + t 



i+(x + t)</> + (y + t)dt. (3.13) 



This defines a continuous linear operator on L 2 (0, oo), such that the compression to 
L 2 (x, oo) is Hilbert-Schmidt for x > 0. 

Further, r^_ defines a continuous linear operator on L 2 (0, oo) since the kernel is a 
sum of Schur multiples of the Carleman operator, namely 

/cosxcosy sinxsinyx /sinxcosy cosxsinyx 

+ = ki( + M + • 3.14 

V x+y x+y J \ x+y x+y J 

With f(x) = e ix and 

M = *r™ (X16) 

we have by Theorem 2.1 of [3] the identity 

f(x)g(y) - f(y)g(x) = f°° e^^dt 
x-y J (x + t)(y + t)' 

Q(viii) Now we consider (3.11) when q 2 = b 2 = 1, and bi = —2(n + 1). Then 
g(u) = u(p(2~ 1 u) satisfies Laguerre's equation 

g"(u)+(^ + ^±±)g(u) = 0, (3.17) 



4 u 

so that, when n + 1 is a positive integer, g{u) = ue~ u ^ 2 Ln\u) and <p(u) = e~ u Ln\2u) 
gives a solution. See 2.2 of [3] for the corresponding Tracy-Widom operator. 
This gives a complete catalogue of the possible quadratic cases. 

4. Hyperbolic cases 

In this section we consider the case in which a and b are hyperbolic functions, giving 
some L, and obtain V in terms of standard special functions such that L and V commute. 

Without loss of generality we can choose t = 2, since other cases occur by rescal- 
ing. The change of variables x = e~ 2u gives a unitary transformation £ 2 ((0, oo); du) — > 
L 2 ((0, l);dx/x), and we modify the definition of the Hankel operator accordingly. 

Definition (Hankel operator). For p E L 2 ((0, l);dx/x), the Hankel operator with kernel 
p(xy) is 

T p h(x)= f p ( xy )h(y)^-, (4.1) 

Jo y 

where h is in some subspace of £ 2 ((0, l);dy/y). 

Lemma 4.1. Suppose that a is hyperbolic, so a(x) = hi cosh to; + h 2 sinhte + /i 3 where hi 
and h 2 are not both zero. Then f3 of Theorem 2. 1 exists if and only if b(x) = /i 4 cosh tx + 
h$ sinh tx + h^ for some constants and then 

h 4 sinh tv /2 + h 5 cosh tv/2 
^ ~ hi sinh tv /2 + h 2 cosh tv /2 ' ^ ' ' 
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Proof. This follows from Lemma 2.2 since we need to find (3 and b such that 
b(x) - b(y) 



smht(x — y)/2 



2 (hi sinh t (x + y)/2 + h 2 cosh t (x + y) /2)f3(x + y) . (4.3) 



Hence b"'(x) = Bb'(x), and of the types (Q), (C) and (H), only a hyperbolic b has the 
right form with v = x + y. 

□ 

Proposition 4.2. (i) With t = 2 and x = e~ 2u , the differential equation (2.2) for <j) 
becomes in the new variable 

d 2 (p | 2(h 2 - hi) d(p h 4 + h 5 + (h 5 - h 4 )x Q 

dx 2 h! + h 2 + (h 2 - hi)x dx kx 2 {h x + h 2 + (h 2 - h\)x) ' 

and the commuting differential operator L becomes 

—^(2x{(h! - h 2 )x 2 + 2h 3 x + (fit + ^2)}^) + (|(/i 4 - h 5 ) + h 6 + ^-(h 4 + h 5 )^jf. 

(ii) The equation (4.4) may be reduced by change of variables to the hyper geometric 
equation or the confluent hypergeometric equation. 

Proof, (i) We have 

htil - x) + h 2 (l + x) v h v ' 

and by Lemma 4.1 we have 

P= k h f~ X \ + I ! 5i ' + X \ (0 <*<!), (4.6) 
tuil-x) + h 2 (l + x) 

so (2.2) reduces as stated. One obtains the formula for L by changing variables. 

(ii) In 4.3 below, we consider the nature of the singular points of the differential 
equation. Effectively there are four constants in (4.4), namely hi ± h 2 and /14 ± /15; by 
taking /i 3 = —hi, we ensure that a(0) = 0. One can easily verify that the effect of the 
change of variable x = 1/y is to preserve the shape of the formula (4.4) and to interchange 
the constants hi + h 2 <-> h 2 — hi and h 4 + h$ <-> /15 — ^4. 

In cases H(i), H(iii) and H(vii) below, there are two regular singular points, so the 
solution is elementary; in cases H(ii) and H(iv), the singular points are zero and infinity, 
and one of them is irregular, so the equation is confluent hypergeometric type; whereas 
in the remaining cases H(v) and H(vi), the three singular points are all regular, so the 
equation is of hypergeometric type. 
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□ 



Examples 4.3 (Hyperbolic cases). 

H(i) If hi = h,2 and h± = /15, then the differential equation (4.4) reduces to 



*"(«)- ^^) = 0, (4.7) 



which has solutions 0(w) = w p where Ap(p — 1) = h±/h\. When p > 0, the corresponding 
Hankel operator is Hilbert-Schmidt. 

H(ii) If h\ = hi and /14 7^ /15, we obtain after rescaling 

/, («) + i^-0(«) = ±i0(«), (4.8) 

with z/ > where the solutions are 

<j>+{u) = yftiK„{2^u), cj)_{u) = ^J u (2^). (4.9) 

Indeed, emerges for the choice hi = hi = — 1, /13 = 1, /14 = 5 — v 2 and h$ = — 3 — z/ 2 ; 
whereas 0_ emerges for hi = hi = —1, /13 = 1, /14 = — 3 — v 2 and /15 = 5 — z/ 2 . We have 

X < o-l_l/2..1/4„-2,/S „ .. . ( 4 - 10 ) 



2- 1 7r 1 / 2 -u 1 / 4 e- 2 v^ as u -> 00, 

so r<^ + defines a Hilbert-Schmidt operator on L 2 ((0, l);<ia;/x) when z/ < 1. 
Further, by [5, p. 436] we have 

, f , J I> + l)-V 1+ ^/ 2 as«^0+ ..... 

< ^- W ^\7r- 1 /V/ 4 cos(2 v ^-^-f) as«^oo; ^"^ 

so r^_ defines a Hilbert-Schmidt operator on £ 2 ((0, 1); dx/x). The associated Tracy- 
Widom operator on L 2 ((0,l);dy/y) has kernel 

^WMi^MW = f M2 ^ W2VFy)dtt (412) 

•'<<' - V Jo 
and the commuting differential operator is 

WW - -,)*) + (-4, + i^! - 2,)/ W (4.13) 

with boundary conditions /(0) = /(l) = 0. In random matrix theory, this is associated 
with hard edges, such as occur with the Jacobi ensemble. 
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H(iii) If hi = —h 2 and h$ = —h^, then we obtain 

fW + VW-^W=0, (4-14) 



so we have solutions <f>(u) = u p where 4p 2 + Ap — h 5 /h 2 =0. 

H(iv) If hi = —hi and h& 7^ — /15, then we change variables to <j>(x) = ip(y) where 
y = 1/x and obtain 

d 2 tfj fh 4 + h 5 h 5 -h A \ 



dy 2 V 8hxy h ± y 2 , 
As in case H(ii), we thus we obtain solutions 

Mx) = 7^ K ^' - ( " ) = ^(^)- 

By (4.10), 4> + gives a Hilbert-Schmidt Hankel operator on £ 2 ((0, 1); dx/x); whereas, by 
(4.11), </>_ gives a Hankel operator which is not Hilbert-Schmidt. 

H(v) When h\ 7^ 0, hi = and h± = —h§ 7^ 0, the equation (4.4) reduces to 
hypergeometric equation. In particular, if hi = 1/4, hi = 0, /14 = — /15, and fi and z/ 
satisfy \i + v = 1 and /uz/ = — 2/14, then we have (1.4) with A = and nonzero parameters 
H and v; however, the usual series for the hypergeometric function F(fi, v, A, x) is then 
undefined. By an identity from [4, p. 1073], the function 

(t>{x) = xF(fj,+ l,v+l,2,x), (4.16) 

gives a power series solution which is analytic for \x\ < 1, with 0(0) = 0, and diverges 
everywhere on the circle {z : \z\ = 1} by [4, p. 1066]. 

H(vi) In the generic case hi 7^ ±h 2 and h& 7^ +h&, we introduce the regular singular 
point ( = —(hi + /i2)/(^2 — hi) and parameters ai,a2,(3i and fa by the simultaneous 
quadratic equations 

h 4 + h 5 h 5 -h 4 
aiOL 2 = — — rr, P1P2 



4(/ii + /i 2 ) A(h 2 -hiY 
«i + a 2 = l, Pi +fo = l. (4.17) 
Then the differential equation (4.4) is given in Riemann's notation [12, 5] by 

00 c 

<j){x) = P{ a x fa -1; x }, (4.18) 
0L2 P2 
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and hence by [5, p 156], may be reduced to the hypergeometric equation (1.4). 

H(vii) Finally, when h± ^ h 2 and h± = h$ = 0, as in Q(iii) we have elementary 
solutions 

<f>{x) = C 1 + ° 2 — . (4.19) 

5. Circular cases 

In this section we consider the remaining case, namely when <p, a and b are circular 
functions. The results of this section are quite analogous to those of H(v) and H(vi); 
although they are somewhat contrived, since the notion of a Hankel integral operator over 
the circle is not in common use. 

Suppose that a, b and <fi have period 27r/i, and that a(0) = a(2n/t) = 0. We extend 
/ G C£°(0, oo) by f(x) = for x < 0, and introduce F(y) = E^l-oo f(v + ^irk/t), which 
is 2iv/t periodic. Evidently 

/ <P(x + y)f(y)dy= / <f)(x + y)F(y)dy (5.1) 
Jo Jo 

is also 2-K/t periodic. 

Definition (Hankel operator). The Hankel operator on L 2 ((0, 27r/£); dy) with kernel 0(a;+ 
2/) is 

r2ir/t 

TF{x) = / </>{x + y)F{y)dy. 
Jo 

We consider T as an operator on C°°(0, 2n/t) : and look for a second-order differential 
operator L as in (1.2) such that LT = TL. 

Lemma 5.1. Suppose that a(x) = c\ costx + c 2 sintx + C3 is circular, where c\ and c 2 are 
not both zero. Then f3 of Theorem 2.1 exists if and only ifb(x) = c 4 costx + c 5 sintx + c 6 
for some constants and then 

, , -c 4 sintM/2 + c 5 costw/2 

p(u) = ; — . (5.2) 

—c\ sint-u/2 + C2 costu/2 

Proof. This follows from Lemma 2.2 since we need to find f3 and b such that 
b(x) - b(y) 



sint(x — y)/2 



21 —c\ sint(x + y)/2 + c 2 cos t(x + y)/2y(x + y). (5.3) 



Hence b"'(x) = Bb'(x), and of the types (Q), (H) and (C), only a circular b has the right 
form. 
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□ 

Proposition 5.2. (i) Let r = tan-u and t = 2. Then differential equation (2.2) for <j) 
becomes in the new variable 

d 2 (j> 2ci d(p -c 4 r + c 5 0(r) _ Q , g ^ 

<ir 2 C\T — C2 dr — C\T + c<i (1 + r 2 ) 2 

and tie commuting differential operator transforms to 

"(I + ^(Wl " - 2 > + **r + cs(l 4- +(^ + 0^ + <*)/■ 

(j'i) Tie equation (5.4) may be reduced by change of variables to the hyper geometric 
equation. 

Proof, (i) This follows from Theorem 2.1 and Lemma 5.1 by calculation. 

(ii) The differential equation has regular singular points: at c^jcx, when c\ ^ 0; at 
oo, when c\ = 0; and at ±z, when C4 7^ or C5 7^ 0. When 04 = 05 = 0, the equation has 
elementary solutions, as in H(vii) and Q(iii). The effect of the change of variable r = 1/s 
is to preserve the shape of (5.4) and to interchange the constants c\ <-> C2 and C4 <-> C5. 
By [5, p. 156], the differential equation may be reduced by change of variables to Gauss's 
hypergeometric equation. 

□ 

Examples 5.3 (Circular case). We present solutions of the differential equations in the 
special case where the singular points are 0, ±i. Let c\ = — 1, C2 = 0, C4 = 3 and C5 = 0; 
so that, the differential equation in the original variables is 

(j)"(x)+2cotx<p'(x) + 3(j)(x) =0, (5.5) 

which has general solution 

4>{x) = cj cosx — cs(cosecx — 2 sins). (5-6) 

The Hankel operator on L 2 ((0, 7r); dx) with kernel cos(x + y) has rank two and eigen- 
functions cosx and sin a;. In the new variable, the solution is the algebraic function 

,/ x c 7 r + c 8 (t 2 - 1) 

^ T > = ttTt — • 

rVr + 1 

Proof of Theorem 1.1. The Propositions 3.2, 4.2 and 5.2 cover the three cases that 
together give the proof of Theorem 1.1. 
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□ 



6. Singular numbers of Hankel integral operators 

Let be the Hankel operator on L 2 (0, oo) with kernel <j)(x + y). In applications to 
random matrix theory, the ultimate aim of the analysis is to prove properties of det(J— T 2 ). 
If W = T^, and W has eigenvalues Ai > A2 > . . . listed according to multiplicity, then the 

1 11 

singular numbers of are Sj = A • for (j = 1,2,...). In this section we prove results 
which show that if <p is analytic on a suitable domain and satisfies various growth bounds, 
then r<^ is a trace-class operator and that its singular numbers are of rapid decay. An 
important feature of the results is that if <p{z) satisfies the hypotheses, then the translated 
functions (p s (z) = <fi(s + z) also satisfy them for all s > 0, up to modified constants. This 
reflects the fact that the compression of r 2 to L 2 (s, 00) is unitarily equivalent to r 2 s on 
L 2 (0,oo). 

Definition (Hankel matrix). Suppose that (7j)?^=i is a sequence in £ 2 (Z + ). Then the 
densely defined linear operator T with matrix [7j + fc_i] with respect to the standard or- 
thonormal basis of £ 2 (Z + ) is a Hankel operator. 

Theorem 6.1. Let cj> be an analytic function on the half plane {z = x + iy : x > —5} for 
some 5 > 0, and suppose that there exists e > and M e {8) > such that 

\(p(z)e £z \ < M e {8) (Sfcz > -5). (6.1) 

(i) Then has singular numbers Sj such that 

fsj ^0 (j - 00) (6.2) 

for all integers p > 0. 

(ii) For all integers p > 1, there exists C p > such that 

logdet(J + xYl) < C p x 1/p (x > 0), (6.3) 
and the entire function det(I + zY 2 ^) has order zero. 

Proof, (i) The operator with kernel e~ 1 (p(e~ 1 (x+y)) is unitarily equivalent to the operator 
with kernel <f>(x + y), and e~ 1 (p(z/e) satisfies (6.1) with e = 1, M 1 (e5) = e~ 1 M £ (5) on 
{z : $tz > —e5}. Hence we assume without loss of generality that 4> has been rescaled 
so that it satisfies (6.1) with e = 1. Let h n (s) = e~ s : Ln\2s), so that (/ij)°^ gives 
an orthonormal basis of L 2 (0, 00), and is associated with a unitary equivalence between 
L 2 (0, 00) and £ 2 (Z + ). Then the Hankel integral operator on L 2 (0, 00) is unitarily 
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equivalent to the Hankel matrix T = [lj+k-i\fk=\ on ^ 2 (^+), where 7-,- = J °° 4>(x)hj(x) dx 
as in [6]. We shall show that the entries of V are of rapid decay as j + k — > 00. 
By integrating repeatedly by parts, we find that 

/•OO 

7j = / (p(x)hj(x) dx 
Jo 

= Sr/ (^(^))y e - 2 ^ (6-4) 

where by (6.1) e z (j)(z) is analytic and bounded inside the circle with centre x and radius 
x + 5. Applying Cauchy's estimates, we obtain 



x 3 



di 



dx 3 



x \ 



(e^(x)) <j\ Ml (5)(—) (6.5) 



and hence 



tel£MlW /„ VTW e " Idx ' (6 ' 6) 

and summing this estimate over j we obtain by the monotone convergence theorem 

poo 

/ (x + S) p+2 e- 2x dx, (6.7) 
Jo 



< M 1 {6) 
~ Sp+ 2 



where the last step follows from the binomial theorem. 

We approximate V by the Hankel matrix rV = [lj+k-i^{{j,k):j+k<N}]i which is zero 
outside the top left corner and has rank less than or equal to N. By considering approxi- 
mation numbers [7], we find that 

sn < ||r - r A r||s(£2), (6.8) 

and since the norm of a matrix is smaller than the absolute sum of its entries, we deduce 
that 

s N < Yl 3hj\ 

j=N+l 

00 

<N-pj2f +1 h\ 

3 = 1 

S^§F((^ +2 + r(P + 3)), (6.9) 
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where we have used (6.7) at the last step. We can repeat the argument with p+1 instead 
of p, and hence deduce that j p Sj — > as j — > oo. 

(ii) We introduce the counting function n(t) = §{j : ts 2 > 1}, and observe that by 
(6.9), 

n(t) < c p t 1/p (t > 0) (6.10) 

for some c p > 0, since s 2 < c p j~ p for all j and n(t) = for < t < ||r^|| -2 . A standard 
summation formula then gives 

oo 

logdet(J + xTl) = log JJ(1 + xs]) 

i=i 

f 00 n(t)dt 
~ X Jo t(x + t) 

< c p x 1 / p 7Tcosec (n/p). (6-11) 

This gives the asserted bound on the growth of log det(/+xr 2 ), and shows that det^+zT^ ) 
has order zero. 

□ 

The following applies to Q(vi), Q(vii) and Q(viii) of Examples 3.3. 

Corollary 6.2. Let 4> be either: 

(i) e- x L { ^(2x); 

(ii) (x + s) _1 e _<x+s ) ; or 

(Hi) (x + s)~ 1/2 K v (2^/x + s) where s > 0. 

Then the eigenvalues of the Hankel operator on L 2 (0, oo) are of rapid decay, so j p Sj — > 
as j — > oo for all integers p. 

Proof. Theorem 6.1 applies directly to <j> in cases (i) and (ii), and we can adapt the proof 
of Theorem 6.1 to deal with <fi in case (iii). Let z = re 10 and < 5 < s. One can show 
that there exists a constant C such that 

\(z + s) 1/2 K u {2^fz~T~s)e z \ < C(s - 5)~ 1/2 e r ^s-(rco S e)^ > -5), (6.12) 

and hence that 

\lp{z)e Z \ < C(S - 5)-l/2 e 2x+«5-(2x+,5) 1 /2 ^ ^ 

when z lies on the circle with centre x and radius x + 5. Now we can apply Cauchy's 
estimates as in (6.5), and then follow the proof of Theorem 6.1. 

□ 
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Let the Fourier transform of / be Ff(i) = e f(x)dx/V2n. 

Proposition 6.3. Let <j> be an analytic function on the strip {z = x + iy : \y\ < a} and 
suppose that there exist K, 5 > such that \<j>(z)\ < Ke~ s ^ for all real x and \y\ < o. 
Then there exist C\ and K2 > such that the singular numbers of the Hankel operator 
with kernel (f)(x + y) all satisfy 

SN^Kde-^ 3 . (6.14) 

Proof. By shifting the line of integration in the Fourier integral, one can show that Fcj) 
is analytic on the strip = £ + ii] : \rj\ < 5}. Further, there exist C 3 and e > such that 
\F<I>(£ + irj)\< C 3 e- £ ^l for all £ + in on this strip. 

The Fourier transform of h n is easy to compute, and by Plancherel's theorem we have 

poo ■ roo (P-i\ n 

ln = I cf>(x)h n (x) dx= — I F<Kt) ,,, (6-15) 



27r7_oo (£ + *) 

We shift the line of integration up to = £ + id : — oo < £ < oo}, and split the range of 
integration; the middle range of integration [— £o, £o] is dominated by the rational factor, 
where we observe that 

«+«-•>• ^i±^tr (ki<&); (6.i6) 



« + « + - \Q + (1 + 6)2 
meanwhile, the contributions from |£| > £ are bounded by 



-oo 



-So r°° 



Uo (t + *S + i) n+ 
In particular, with £o = n 1 ^ 3 , we obtain the bound 



l7n 



^cJ ^p% Y /2 A---'\ (6.18) 
Vn 2 / 3 + (1 + 5) 2 / e v ; 



2 / 3 + (l + 5) 



Due to a similar approximation argument as (6.7), the singular numbers of consequently 
satisfy (6.14). 

□ 

This result applies to an orthonormal sequence of functions which is used to analyse 
the Gaussian orthogonal ensemble as in [1] . 

Corollary 6.4. Let 4> n be the n th Hermite function. Then the singular numbers of the 
Hankel operator T^ n on L 2 (0, oo) with kernel (p n (x + y) are of rapid decay as in (6.14). 
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Proof. The hypotheses of Proposition 6.3 hold for 4> n {z) = H n (z)e z2 / 4 : where H n is the 
Hermite polynomial of degree n. 

□ 
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